[Improve][Connector-V2][Doris]Refactor some Doris Sink code as well as support 2pc and cdc #4235
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zy-kkk
force-pushed
the
doris_sink_2pc
branch
from
826bde8
to
e2349d8
Compare
zy-kkk
changed the title
CalvinKirs
reviewed
Mar 3, 2023
...doris/src/main/java/org/apache/seatunnel/connectors/doris/sink/committer/DorisCommitter.java
Outdated
Show resolved
Hide resolved
CalvinKirs
reviewed
Mar 3, 2023
...v2/connector-doris/src/main/java/org/apache/seatunnel/connectors/doris/rest/RestService.java
Outdated
Show resolved
Hide resolved
CalvinKirs
reviewed
Mar 3, 2023
...v2/connector-doris/src/main/java/org/apache/seatunnel/connectors/doris/rest/RestService.java
Outdated
Show resolved
Hide resolved
CalvinKirs
previously approved these changes
Mar 7, 2023
...or-doris/src/main/java/org/apache/seatunnel/connectors/doris/sink/writer/DorisSinkState.java
Outdated
Show resolved
Hide resolved
...r-doris/src/main/java/org/apache/seatunnel/connectors/doris/sink/writer/DorisSinkWriter.java
Outdated
Show resolved
Hide resolved
...or-doris/src/main/java/org/apache/seatunnel/connectors/doris/sink/writer/LabelGenerator.java
Outdated
Show resolved
Hide resolved
|
It seems that CI has some problems. Please fix it. |
|
BTW, It is best to add an adapted Doris version |
CalvinKirs
previously approved these changes
Mar 8, 2023
CalvinKirs
reviewed
Mar 8, 2023
|
It seems that has some confilcts. |
CalvinKirs
approved these changes
Mar 8, 2023
Hisoka-X
reviewed
Apr 18, 2023
Comment on lines
+28
to
+48
| /** Serializer for DorisWriterState. */ | ||
| public class DorisSinkStateSerializer implements Serializer<DorisSinkState> { | ||
| @Override | ||
| public byte[] serialize(DorisSinkState dorisSinkState) throws IOException { | ||
| try (final ByteArrayOutputStream baos = new ByteArrayOutputStream(); | ||
| final DataOutputStream out = new DataOutputStream(baos)) { | ||
| out.writeUTF(dorisSinkState.getLabelPrefix()); | ||
| out.flush(); | ||
| return baos.toByteArray(); | ||
| } | ||
| } | ||
|
|
||
| @Override | ||
| public DorisSinkState deserialize(byte[] serialized) throws IOException { | ||
| try (final ByteArrayInputStream bais = new ByteArrayInputStream(serialized); | ||
| final DataInputStream in = new DataInputStream(bais)) { | ||
| final String labelPrefix = in.readUTF(); | ||
| final long checkpointId = in.readLong(); | ||
| return new DorisSinkState(labelPrefix, checkpointId); | ||
| } | ||
| } |
There was a problem hiding this comment.
out.writeUTF(dorisSinkState.getLabelPrefix());but
final String labelPrefix = in.readUTF();
final long checkpointId = in.readLong();Not match.
|
There some problem, you add support for 2pc, but you don't open 2pc in your e2e config. It cause 2pc can't pass with ci. |
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose of this pull request
There were some problems with the existing Doris connector, and I refactor some Doris sink code as well as support 2pc and cdc
Due to insufficient workflow test resources, all Doris tests cannot be completed, so I disable the test class first. I have successfully tested it locally. Later, I will streamline the docker image so that the workflow test can be successful
Check list
New License Guide
release-note.